Method and a system for automatic measurement and tracking of logs, industrial wood and boards

ABSTRACT

A method for tracking and measuring volume, shape and surfaces of objects, such as logs, simultaneously. In series of load and unload operations along a procurement line, series of images are captured by e.g., CCD stereo cameras with sufficient spatial resolution capabilities. Simultaneously, the location of objects (logs) are registered by a GPS system aiding the tracking of the objects. The load and unload operations are typically performed by machineries such as harvesters, forwarders or trucks all equipped with a crane. On such machines digital cameras are mounted on jib arms and a computer system is attached. A GPS system is mounted on the machines too and the computer system is attached here as well. Thus the real time processing of stereo images can be accomplished and the volume, shape and surface of the 3D objects are computed simultaneously with their location in space. The resulting data are sent to a central database that keeps track of objects and their locations. These data are then again available for the following step in the procurement line as “a priory” information, facilitating the computation of the size, shape, surface and location of logs (objects) at the current load/unload operation. A chain of information provided by this invention will profoundly increase efficiency of any production chain subjectable to the presented method.

FIELD OF INVENTION

This invention relates to measurement and tracking of objects by use ofphotogrammetric methods. The invention targets the situation wherebundles of such objects are handled for being collected in largerbundles or for further processing of individual objects. During thisaction the individual objects of a small bundle will be visible to alarge extend. In its simplicity the idea then is by photogrammetricmethodologies to obtain information of a particular object from imageryof one or more small bundles where it appears. More specifically thethree dimensional (3D) extend of an object and properties related to itssurface such at texture and other significant characteristics areextracted. Capturing imagery while grabbing and releasing the smallbundles and/or at successive points during the handling gives theinformation necessary to keep track of the transportation path in timeand space of each individual object. Thus the methodology in totalprovides measurements and tracking of the individual objects.

Clearly the invention encompasses a wide range of possible applications.A generic example is an industrial process where a commodity travelsthrough several steps of subprocesses during the manufacturing. Thespecification here is focused on a unique design for use in the forestryand wood processing industry.

During the last decades forestry has undergone a substantial degree ofmechanization. Today wood is harvested and processed at an increasingspeed and the wood industries makes up large units. Further themachinery used applies highly developed technologies. As a standardfelling machinery and saws at sawmills are equipped with variouscomputer technologies to aid software for optimization in each step ofthe procurement process. At so the same time there is a growing publicawareness of the management of natural resources and the utilization ofthose resources that are harvested. It is generally agreed that futureadvances relating to wood processing are to be obtained from furtheroptimizations of the wood procurement process. The present inventionfacilitates optimizations throughout wood procurement process byproviding detailed information of the wood (objects) at any stage wherethe system is applied.

GENERAL SECTION

The present invention is a real-time computer vision and tracking systemto automatically locate and measure the size and quality of individualhard and softwood logs, pieces of industrial wood, and boards,hereinafter logs.

This system is to rationalize, and increase efficiency and measurementaccuracy throughout the wood procurement process. Further, the precisetracking of the logs allows for accurate documentation in e.g., pursuitof wood certification.

By a computer vision system is understood a single or a series ofcomputer sensor systems with integrated GPS. The vision system locatesand measures the logs at several stages of some wood procurementprocess. The results of one or more of these measurements are to be usedindependently or in conjunction to give precise and unbiased estimatesof position, size and quality of logs at any specific stage.

By tracking is understood a record of the positions of a recognized logalong the procurement process where the system is applied e.g., thepositions of a specific log from felling to any stage along theprocurement process where the system is no longer applied.

The system will satisfy different immediate demands for information ofvarious actors along the procurement process. In particular: i) buyersand sellers demand for accurately measured and classified trading unitsand documentation hereof, and ii) the need to back-track logs and otherwood commodities to their origin in the forest for certificationpurposes.

Further, the information collected by the system up to any given stage(operation) of the procurement process can be utilized for optimizationpurposes in subsequent stages of the procurement process. This increasesthe efficiency of the procurement process.

A schematic representation of the wood procurement process is shown inFIG. 1. The action pattern of logs being loaded and unloaded by similardevices e.g., cranes, is repeated at all operations throughout theprocurement process. Therefore the system is designed to apply similarcomputer sensor systems and similar image analysis algorithms at allstages.

Typically logs are traded at one or more of the operations listed inFIG. 1. Thus the unit for trade is a varying collection of logs and thesystem is designed to compute aggregate values for such collections oflogs.

The system can be applied at any action or set of actions e.g., anoperation listed in FIG. 1. In consequence it is the responsibility ofbuyers and sellers to decide on what stages it is appropriate to applythe system. Back-tracking boards to their origin in the forest doesnaturally require the system being applied at most unload and loadpoints (actions).

The present invention suggests measuring individual logs using a stereovision system integrated with GPS. The GPS system provides the locationin space of the equipment that handles the logs and the vision system.Combined with automatic recognition of the logs in imagery captured bythe vision system the path of each individual log can be mapped toprovide tracking of the logs.

The core idea of the vision system is that it captures imagery of thelogs during load/unload actions. During these actions only a few logsare handled at a time by a crane or similar device. Imagery of a bundleof few logs allows for a complete recognition of each individual log inthe bundle. Candidate mount points of vision system are e.g., grab onharvester or forwarder, body of harvester or forwarder, truck at plant,conveyer belt or any other place where the logs can be seen from visionsystem.

Further, imagery of the stacks (collections of logs) that are beingloaded or unloaded are captured continuously to monitor exactly whereeach bundle of logs are placed or taken from. In this way stacks areconsidered cohorts of individual logs with their individual propertiessuch as transportation path and size. Thus aggregate quantities anddistributional statistics at the stack level can be computed tocharacterize a stack.

The idea of measuring logs on the fly by remote sensing and stacks beingcohorts of logs with an associated record of information at the loglevel is an advance compared to todays practices. Common practices is tomeasure stacks in a separate process and characterize stacks by theirouter measure and other aggregate quantities.

Each individual log is being monitored several times during theprocurement line. At any load/unload action stereo imagery of each logwhile located in the unload stack, the crane and the load stack is beingcaptured. This comprehensive information source forms the basis of animage analysis algorithm to estimate the size and location of the logswith a high level of accuracy. Integrating imagery across severaloperations in FIG. 1 in the image analysis adds to the level of accuracythat can be obtained.

The image analysis algorithm is designed to adopt information from manysources. Hence the algorithm allows data fusion from sources such asfield inventory, harvester, regional prior information on size andquality, and information dynamically gathered during a forest operationto ensure self-calibration.

An important by-product of the real-time implementation of the system isthat the influence of decay and other biological factors can bequantified. Especially if there is a time-lag between the creation of astark and it being picked up, using the system at both of theseoperations makes it possible to monitor changes in the stack.

The present invention is considered applicable for other industrial useswhere 3D objects are being handled in a similar repeated load/unloadfashion in stacks.

SPECIFIC SECTION

The system is designed to be build from standard components thatcomplies with industry standards to ensure portability and low cost. Inparticular standard cameras, standard image formats, standard graphicshard and software, standard computer units with accompanying operatingsystems suffice to build an operational system.

One example of an actual implementation of the this system is based onCCD digital cameras with a spatial resolution of 2-5 mm² per pixel forobjects located at 2-15 m distance in physical space from the visionsystem. The image format is compressed tiff and the softwareimplementation of this system written in C/C++/C# runs a windows orLinux box. The graphics procedures runs through an accelerated 3Dgraphics card using the OpenGL language.

In the following is by machinery understood some machinery that handleslogs along the wood procurement process. The device used by suchmachinery to handle the logs will be referred to as a grab.

Some examples of machinery are Valmet and Ponsse harvesters andforwarders, Mack and Volvo trucks with trailers build to load logs andequipped with jibs, and genuine cranes at the saw or paper mills.

The stereo vision system is to be mounted on the machinery such that acontinuous series of images of the logs that are being handled can becaptured. The ideal strategy is to capture one or more set of images ofthe load and unload stacks and of the bundle of logs in the grab foreach bundle of logs being handled.

As an example harvesters have lights mounted pairwise on their jibs armsand shielded by a frame. A set of cameras are conveniently mounted nextto the lights within these frames. When the operator fells a tree and ateach time a log is cut off the vision system takes images. For aforwarder the cameras are mounted similarly, but here the images aretaken whenever a new cohort of logs i grabbed or released and at regulartime intervals in-between or whenever the orientation of the cohort inthe grab is optimal relative to the cameras.

The GPS unit is mounted on the machinery too. From its continuouslogging of signals the absolute orientation and location in space of themachinery is known. Given the orientation of the grab relative to themachinery the location in space of the grab is computed.

Logs are three-dimensional (3D) solid objects. In consequence the imageanalysis algorithms are implemented by use of software routinessupported by standard 3D graphics hardware. Computing images of avirtual 3D universe that represents the real world stacks and bundles of3D logs then is fast and straight forward.

The resulting information about stacks or bundle of logs computed fromthe images are subsequently stored in a central database. Thisinformation include basicly estimates on size, location and quality ofthe individual logs. Further, aggregate values at stack level as well asimage scenes (movie pictures) of part of or of the whole recordedoperation should be stored in the database too.

The central database can be located on any pre-selected machinery oroffice computer. The only requirement is that the other computers areable to access the central computer by an online connection e.g., theInternet by a phone card or a similar standard device. This way thesystem is robust against hardware failure on any link in the procurementline.

The image analysis algorithm for tracking and size estimation isconceptually split into two stages. Stage 1 is a coarse recognition ofthe logs taken from or added to a stack, and the logs held in the grab.Stage 1 is accomplished by template matching or other filtering andprovides approximate location and size of logs in the virtual 3D world.Stage 2 is to infer accurately about each individual log from the imagedata available and other prior information. The approximateconfiguration of the virtual 3D world from Stage 1 is used as initialvalue for stage 2. Stage 2 is more involved and applies a formalstatistical analysis. The outcome of the Stage 2 axe configurations ofthe virtual 3D world (one for each captured image used in the analysis)that describes the real world best given the information available. Inother words, Stage 2 provides the transportation paths of the logs andtheir size and quality.

Both of Stage 1 and Stage 2 require a specification of the virtual 3Dworld that mimics the real world. That is, a mathematical model is setup to describe the 3D shape of the logs and their transportation path.This model is then adjusted (estimated) to fit best possible with theactual taken images.

A possible mathematical model for a log is that the stem center followsa 2nd, 3rd or 4th order polynomial (in a plane) and that cross-sectionsperpendicular to the stem center are circular with diameter a linearfunction of distance to the stem base (FIG. 4). Alternatively diametercan be specified by stem taper functions commonly described inliterature.

Given a specification of the virtual world the mapping into image space(image formation) is required to compute the images of the virtualworld. It is by comparison of these computed images and the actual takenimages that the mathematical model can be estimated.

As an example if the vision system is based on CCD cameras the mappingfrom object space (the physical world) into image space is thecomposition of a central projection and the deterioration by the lensand irregularities induced by the chip. The two latter together makes upthe so-called inner orientation and is a specific property of eachcamera. The inner orientation is established separately from a testscene in a laboratory. The location and attitude of the cameras in thevision system relative to one-another together with the model for imageformation makes up a stereo vision system. That is, a system that allowsfor 3D reconstruction of the objects of interest (the logs).

The Stage 1 filter that singles out individual logs in the grab operateson a complex source of information including one or more of: grab“width”, location of grab in space (implicitly orientation of logs),exact record of the logs handled if taken from stack already monitoredby the system, other prior information on log size e.g., from harvester.

The Stage 1 filter that singles out individual logs taken or added to astack operates on successive images of the stack to detect changes inits surface. Candidate locations of logs are identified from thesechanges (FIG. 3). Information about the location of the grab where ittook or added logs and the bundle that was handled is used in support ofthis filter.

In essence the Stage 1 filters map the complete transportation path ofeach individual log and the tracking is completed. Note that image dataand GPS data are used in conjunction to complete this task.

The inference framework for Stage 2 is Bayesian where the likelihoodterm f(I;θ) is the density of the image data I under a statistical modelparameterized by θ and π(θ) is the prior on θ. The statistical modelencompasses the model of the virtual 3D universe, the image formationprocess and the randomness of noise in the imagery. The posterior p isgiven by p(θ;I)∝π(θ)f(I;θ). Since θ includes the parameterization of thevirtual 3D world, maximizing p provides an estimate of the configurationof the 3D real-world logs. A natural choice of the prior π is knowledgeabout the size distribution of the logs under study. Typically thisinformation is easily obtained from growth models or yield tables.

Technically each log in the virtual world is represented by a discretespatial object during the maximization of the posterior. The set ofpoints that spans the log are derived from the underlying mathematicalmodel. The triangulation between these points then makes up the surfaceof the log. Let θ′ be the parameter that parameterizes a particular log.Then θ′ is a subset of θ. Having maximized p with respect to θ thereforeprovides the best value for θ′. Volume and curvature and other importantproperties of the log are therefore best computed from the value of θ′.

Since logs axe solid objects with a certain regularity in shape, theparameter θ′ can not be considered a completely free parameter. Thus theposterior p is maximized under suitable smoothness constraints on theshape of logs. Further different logs are in principle not allowed tooccupy the same physical space. In practice this constraint is relaxed abit to accommodate for the fact that θ′ is not a perfect representationof the real-world logs.

If the system is applied at several stages along the wood procurementline, the image data I is the aggregate image data from all theconsecutive images captured. This implies that the 3D configuration ofeach log is estimated from image data captured from different directionsrelative to the log minimizing the occluded volume of the logs.

The system operates in absolute units in the sense that it produces themeasured sizes in meters or some other absolute unit. Technically thisrequires the scale be known in the imagery captured by the stereo visionsystem. In recognition of the fact that the system operates in adisturbed environment, marks on the grab or other parts of the machineryis used in addition to a known base line of the vision system to obtainscale.

The Stage 2 image analysis gives a coarse assessment of wood qualitye.g., based on curvature of a log and the appearance of growth rings atcross sectional cuts of logs. Additional image analysis routines areapplied to measure rot percent, knots, bark texture changes to get asmuch information about wood quality as possible.

When a static of logs is completed the system computes aggregatequantities for the stack. The set of quantities computed may be changeddynamically by the user. Typical aggregate quantities are: number oflogs in the stack, total volume of the logs in the stack, and size andquality distribution of the logs in the stack.

Whenever logs are collected on some machinery and transported to someother location for unload the real-time requirement on the imageanalysis algorithms can be relaxed. It is not until the unload actiontakes place that the system must provide the image analysis results.This fact is particularly useful in the first stage where the system isapplied is on a forwarder in the forest. At the first stage no priorinformation about the log is available and successful log recognitionand size estimation may require an extended computation time. Usage ofthe system at subsequent stages along the procurement process willrequire less computer run time because detailed prior information aboutthe logs are available.

The vision system is mounted on the machinery so that it can captureimages of both the grab and the load and unload sta. The ideal situationis that the system be mounted such that for each bundle of logs that isbeing handled images of both grab and stacks can be captured. Somecandidate mount points are jib arm or the body of the machinery as shownin FIG. 2.

A unique feature of the system is that the measuring device, i.e. thevision system, is not in contact with the logs. This implies that thesystem is less prone to deterioration, and thus the demand forcontinuous calibration of the system while being applied is low. This isan advantage to measuring devices that are mounted in the felling deviceof a harvester.

The present invention suggests both the tracking and the size estimationtasks solved by the system. As time progresses and e.g. bio-informaticstechnologies evolve, the tracking task may be completed from the DNAfootprint of each log or some other recognition procedure. By virtue thesystem is designed to incorporate such information in the Stage 2 imageanalysis. In other words, the system is designed to be complemented byother information sources that aid the tracking and size estimation.

The most critical factors to the system are the sensor conditions andthe number of logs that the grab handles. Operating the system duringe.g., night time with a sensor sensible to visible light thus requiresthe logs being lighted by some artificial light source. If a particularlog is occluded by other logs when located in the stack or in the grab,the system can only provide an approximate estimate of its size. Thesolution space for the volume occupied by that particular log is howeverquite small so that the overall performance of the system i notadversely effected.

If the order of logs in a stack is modified by some outside factor orits geometric constitution has changed much, the change is recognized bythe system and it starts monitoring if logs are missing in the stack.

1. A method of tracking objects among a plurality of like objects beingtransported from a geographical location, the method comprising thesteps of determining the geographical location of each individualobject, taking at least one image of the object, determining, from theat least one image of the object, characteristic data about each objectenabling identifying the object, transporting a plurality of objects,whose geographical location and characteristic data have beendetermined, from their geographical location to a handling station,handling, at the handling station, a bundle comprising at least one ofthe objects and with at least a part of each object being visible, thehandling including taking at least one image of the bundle of objects,and identifying, from the at least one image of the bundle of objects,each object in the bundle.
 2. A method according to claim 1, wherein theobjects are logs.
 3. A method according to claim 2, wherein thegeographical location is the location of harvesting.
 4. A methodaccording to claim 2, wherein the characteristic data includes thevolume of each individual log.
 5. A method according to claim 2, whereinthe characteristic data includes the mass of each individual log.
 6. Amethod according to claim 2, wherein the characteristic data includesthe quality of each individual log.
 7. A method according to claim 2,wherein the characteristic data includes the species of each individuallog.
 8. A method according to claim 2, wherein the characteristic dataincludes curvature of the log.
 9. A method according to claim 2, whereinthe characteristic data includes taper of the log.
 10. A methodaccording to claim 2, wherein the handling includes unloading theplurality of logs from a forwarder.
 11. A method according to claim 2,wherein the handling includes reloading the plurality of logs onto aforwarder.
 12. A method according to claim 2, wherein the handlingincludes processing the logs in a sawmill.
 13. A method according toclaim 2, wherein the geographical location of each individual object isdetermined using a satellite based global positioning system.
 14. Amethod according to claim 1, wherein the at least one image includes apair of stereo images taken by a pair of cameras.
 15. A method accordingto claim 14, wherein the pair of cameras is mounted on an equipmenthandling the bundle of objects.
 16. A system for tracking individualobjects among a plurality of like objects being transported from ageographical location, the system being adapted to perform the methodaccording to claim 1.